A Performance Evaluation of Text Analysis Technologies by Wendy Lehnert

Beth Sundheim Naval Ocean Systems Center, Code 444 271 Catalina Boulevard San Diego CA 92152 ` Evaluation has become an important and pressing concern for researchers in AI. We need to reassure ourselves and our program managers that progress is taking place, and that our technology is indeed advancing according to reasonable metrics and assessments. The difficulties with evaluating AI systems are substantial and, to some extent, idiosyncratic, depending on the area of specialization. In an effort to evaluate state-of-the-art natural language processing systems, the Naval Ocean Systems Center (NOSC) has conducted three evaluations of English text analyzers during the last five years. This report describes the most recent and most sophisticated of these evaluations, the Third Message Understanding Conference (MUC-3)1 This evaluation was sponsored by the Defense Research Projects Agency (DARPA), which plays a key role in sponsoring evaluations for other types of language interpretation systems, including performance evaluations for speech recognition carried out by the National Institute of Standards and Technology (Pallett 1990).